Feb 23
17
Google is warning against using 404 and other 4xx client server status errors, such as 403s, for the purpose of trying to set a crawl rate limit for Googlebot. “Please don’t do that,” Gary Illyes from the Google Search Relations team wrote .
See the original post:
Google warns against using 403 or 404 status codes for Googlebot crawl rate limiting